ESCAPE main deliverable is wp2 (datalake prototype).
Prototype orchestrator (Rucio) used by ATLAS project (LHC). Data sizes are SKA comparable, but compute requirements are different.
SKA objective: Want to be able to prove that with data remotely located, track it, do stuff with it, manage it.
Rucio test instance has been set up by RAL using IRIS resources. IRIS is a funding body to buy hardware for STFC science. Not easy to procure "person" power.
ESCAPE also has a newer Rucio instance.
May need to deploy our own Rucio instance. Gotten easier. Docker based deployment now easier.
WP2 ahead of schedule. ESCAPE responsible (at the moment) for functional testing, not development.
Abstracts storage level information from user using data identifier (DID). Contains information about where item is stored.
DID has a scope and a name (scope:name), scope is used for project level differentiation. The combined scope:name has to be unique.
Files are grouped into datasets, datasets grouped into containers. Containers can contain datasets and files, and containers.
Movement of data is done by parameterised replication of rules, i.e. this data, one copy, from this location. Rules can have expiration times etc. Multiple rules are evaluated by Rucio, and only the minimum required tasks will be done.
Storage elements are abstracted. This is true to a point, e.g. for HEP. For non-HEP there is a learning curve to the base setup for e.g. host, port, protocol, path.
Relies on a file transfer service (FTS) layer (CERN). If we set up our own Rucio instance, can initially use CERN's file transfer service.
Missing pieces: Permissions/security. Users can see other people's data (although can't edit/delete). Problems with proprietary data. Flat namespace is also a challenge - naming files and finding them in the namespace could be tricky. May need an abstracted, user friendly interface.
Has a Python API, RESTful interfaces.
Part of ESCAPE work package 5.
DIRAC is a workflow management system from CERN. Allows deployable scripts or executing containers across computing architecture. Non-interactive. Provisions compute to execute jobs in batch mode.
Looking at e.g. LSST tech, e.g. JupyterLab. Can deploy JupyterLab on K8s (a container orchestrator) in the cloud.
User connects to Jupyter service -> spawns a service. Another user connects -> replicates service. Scales.
Work at ASTRON to provide an API to connect all the services.
Anna Bonaldi generated some synthetic data for SKA. Participants challenged to identify and classify sources using e.g. AGN etc. 9 participants. Scored and ranked. Pipelines not scalable.
Same as 1, except using a 1Tb datacube. Need a better method to score data (previous was custom, interactive IDL script). Reimplementation of scoring code. Need to set up a cloud machine for generation of dataset.
Could use Jupyterhub to run participant workflows. Also looked at using Kaggle. Well suited for classification rather than identification tasks.